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1.  Introduction.  2WW/DNS  has  developed  worksheets 
and  nP-97  progranmable  calculator  software  to 
facilitate  probability  forecasting  data  gathering 
and  analysis,  and  to  assist  2WW  field  units  in 
probability  forecasting  tasks.  Instructions 
follow  for  completion  of  the  worksheets  and  for 
use  of  the  HP-97  software.  Further  background  on 
this  subject  is  contained  in  AWSP  105-51  and  2WW 

TN  79-1.  (2WV  TNs  78-1.  78-4,  78-6  and  79-6  also 
provide  background  on  probability  forecasting). 

Both  the  Form  26  and  28  are  intended  primarily  as 
individual  forecaster  worksheets.  The  intent  is 
for  forecasters  to  maintain  data  on  their  own 
forecasts  and  to  analyze  the  reliability  and 
sharpness  on  a  periodic  basis  (nor 'ally,  but  not 
necessarily,  monthly).  This  procedure  provides 
forecaster  feedback  which  is  the  primary  purpose 
fair  this  'type" "of  'forecast  verification.  The  com¬ 
putation  of  the  Brier  Score  and  associated 
statistics  as  a  general  rule  should  only  be 
accomplished  for  overall  unit  data.  It  can  he 
done  for  :  <>d  I  v  i  dua  I  s  ,  but  this  is  rn-t  recommended, 
is  a  general  rule  the  tracking  of  skill  scores 
fro.”  month  to  month  is  to  be  minimized  in  favor  of 
time  spent  or  analysis  and  correction  of  the  bias 
in  unit  and  individual  reliability  and  sharpness 
data . 

2.  Use  of  2WW  Form  26  and  2WV  Form  28 

a.  Fo'm  26.  Figure  I  ■  s  an  example  of  a 
completed  Form  26.  This  example  is  from  a  series 
(20)  of  4  category  forecasts.  The  form  can  be 
used  for  2  through  6  categories.  There  are  two 
distinct  sections  on  the  form.  The  first  twenty 
lines  are  used  for  recording  forecasts  (circled 
letters)  and  observations  (check  marks).  The 
last  eight  lines  are  used  for  summarizing  the 
data 

ii'  I"  the  example  on  the  first  1  i tie  a 
ce> forecast  of  0181  is  recorded.  This  fore¬ 
cast  is  reorded  then  by  circling  the  Cat  A  fore- 

.as »  .nder  Pf  a  0*,  the  Cat  B  and  D  under  Pf  = 

10/,  ad  i  ♦  »»  C .  i  f  C  under  Pf  -  80f.  The  obser¬ 
vation  i  ,  ecnrdcd  by  entering  a  check  mark  in  the 
'  I  or  k  .,m_  >  r  the  appropriate  category  lias  been 
>’  l**i  'In  the  example  the  1?  Feh  74  hour  fore- 

.a*  ft -i  i  I  note  that  both  Category  C  and  D  where 

‘ori.*  t  •  .»  40/  In  such  cases  the  category 

./hi  ■  n  /«.»  -dr. i  r  /«••!  .{st  be  annotated  a  lon<j  with 
■  hr  r  0«;  * .  ,j  r  *■  . 

/  '  The  surni.aiy  section  on  the  example 
Ini',  i  r;  if,!  lumbers  i  ri  the  lelt  and  light  margins 
whi-.h  ,i r e  used  here  for  reference.  Data  is  re¬ 
corded  based  on  what  hours  art*  In  be  verified 
(3  ?4  -  the  example),  however,  1  he  suevnary  is 

ace*  --ip  i  •  ,  ‘H*d  for  all  hours  combined.  The  purpose 
f'.-  .  n. '  > t- ,  r  i  n<;  multiple  hour  verification  is  to 
bn  ■  1  •  .?  -’ore  reliable  and  representative  data  base. 

Row  0  If’  each  block  ilong  this  row  the  sum 
•  »»  cat  *»  observation*  (check  arks)  i  •>  tabulated. 
lv<  »e**  a^e  not  indicated.  In  the  example  there 
vri  i'  no  fat  A  observations. 


RowQt^l©©  These  rows  are  the  sumrotion* 
(as  was  row©)  or  the  number  of  observations  for 
the  indicated  categories.  The  purpose  of  rows 
©  through^)  Is  to  facilitate  calculation  of  the 
category  frequencies.  This  Is  done  by  dividing 
each  of  the  sums  on  the  riqht  (next  to  rows 
th rough by  the  total  number  of  observations. 

On  the  example  4  categories  are  used;  consequent!, 
row  5  is  1 1 ned  out . 

RowQ.  The  sum  of  the  observations  (checks) 
in  each  column. 

Row^}  The  sum  of  the  total  number  of  circles 
(forecasts  per  Pf  In  each  column). 

Row(&.  The  quotient  of  O/Tf  or  row^) 
divided  by  row^) 

Row  and  ©.  A  double  check  to  insure  no 
errors  have  been  made  In  the  tabulation.  A 
horizontal  sum  across  rowQwfll  always  total  20 
If  Lbe  sheet  Is  full.  The  sum  of  rpw/7) 1 nd i cated 
i n  (Wy  will  equal  the  product  of  the  value  in  <S> 
times  the  number  of  categories  in  use.  If  these 
two  summations  do  not  check  out  a  tabulation 
error  has  been  made. 

As  a  general  rule  20  forecasts  are  not  a 
large  enough  sample  to  gain  benefit  from  the 
analysis,  consequently,  two  or  three  forms  should 
be  completed  before  the  data  is  tabulated. 

b.  Form  28.  The  primary  purpose  for  this 
Form  is  to  provide  space  for  graphing  and 
analysis  of  reliability  and  sharpness  data.  Data 
can  be  transferred  directly  from  the  Form  ?6  to 
the  Form  28  to  facilitate  this  task.  Space  is 
provided  for  two  sets  of  forecast  data.  See 
Figure  2  for  example. 

(1)  To  plot  reliability  > ho  Pf  vilue  is 
plotted  on  the  vertical  axis  and  the  ratio 
(observed  frequency)  0/Tf  is  plotted  on  the 
horizontal  axis.  Refer  to  AWSP  105*61  I'" 
background  and  instructions  on  the  i nterprotat ion 
of  reliability  dm  a  . 

(?)  Sharpness  Is  a  plot  of  P*  l/eiiical 
ax  i  .)  agjinst  Tf,  the  number  of  forecast-  for 
each  Pf  value.  Again  refer  to  AWSP  1 0  S  I  for 
i ntorpreiai ion. 

form  2fl  can  be  forwarded  to  2WW/0N  for  com¬ 
putation  of  the  Brier  Score  and  associated 
statistics  and  analysis  of  reliability  and  sharp¬ 
ness.  This  data  can  also  be  called  in  by  phone 
if  only  computations  are  needed.  The  data  should 
be  read  In  the  sequence  indicated  on  Figure  3* 

The  data  can  be  punched  into  the  computer  and 
the  statistics  read  back  in  a  matter  of  minutes. 
The  computer  software  and  its  use  is  described 
in  Section  4.  Both  the  form  26  and  28  are  still 
test  forms  and  are  subject  to  minor  modification, 
however,  this  will  not  chanqe  the  way  the  forms 
are  used. 


I 


i  To  jvc  the  HP-97  software  for  computing  the 
Brier  Score,  the  Wrier  Score  Standard,  improvement 
over  standard  (I0S)  and  percent  of  possible 
i mprovarent  over  standard  (PPlOS)  the  following 
instruct  ions  must  he  followed.  The  input  data  is 
all  on  2VU  Form  28.  The  sequential  numbers  in 
bo  Ui  handv*ritinq  on  Figure  3  ore  to  be  punched 
in  the  order  indicated.  In  each  case  below  "enter*' 
means  punch  the  number  into  the  X  register. 
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and  S  are  not  entered  into  the  ■  ale-.'  dor 

Step  p  Enter  121)  The  calculator  will 
automat (ca I K  begin  to  compute  and  will  print  *  he 
Brier  Score  It  ta*es  appro* • mate  I y  10  seconds 
for  the  urogram  to  run. 

Step  6  Enter  122)  through  (26)  the  calculator 
will  prjrt  the  standard,  the  I  OS  and  the  PPlOS  - 
in  that  order.  If  less  than  five  categories  are 
used  zero  must  be  entered  for  each  Cat  frequency 
not  represented;  e.g.  (2*0,  (26)  and  (26)  would 
all  be  zero  in  a  two  category  situations. 
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3  The  sum  of  the  occurrence  in  a  given  category  for  each  Pf.  Standard _ 

Tf  =  The  sum  of  the  circled  forecast  in  a  given  category  for  each  Pf.  IOS _ - 

O'Tf  =  The  quotient  of  the  summed  Os  and  Tf.  .  i.e.  (Ci+02)/(Tfi=Tf2l  PPIOS _ 
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ne  frdiiie  l.ne  of  data  n  each  column  are  keystrokes.  The  left  line  of  data  are  program  sequence  numbers 
'he  r-iqhi  line  of  data  kevstroke  code  figures. 
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